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(54) Title: A METHOD AND A SYSTEM FOR PROCESSING IMAGES 
(57) Abstract 

The present invention relates to a method and a system for processing 
images. The invention relates in particular to a method and a system for 
processing an image using interpolating. The method determines a variation 
value for different sets of pixels in a CCD, and the set of pixels with the highest 
correlation (lowest numeric value) is selected. A missing colour at a given pixel 
is expressed in terms of the information contained in the selected set of pixels. 
If the variation value is below a certain threshold, due to noise or other factors, 
an alternative set of pixels using a different variation method may be selected. 
The missing colour is estimated as a weighted sum of e.g. the green samples 
divided by the weighted sum of e.g. the red samples multiplied by the sampled 
value from the pixel to be estimated. 
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1 ...... 

A METHOD AND A SYSTEM FOR PROCESSING IMAGES 

This invention relates to a method and a system for processing images. The invention re- 
lates in particular to a method and a system for processing an image using interpolating. 

5 

It is well known to represent an image digitally by dividing the image into a large number 
of segments, denoted pixels, and allocating digital values, denoted pixel values, to each 
pixel. Typically, the image is divided into a matrix of rows and columns of pixels and the 
size of a digital image is then given by the number of pixels in a row and the number of 
10 pixels in a column. The pixel values are typically stored in an array in a digital memory. 

For example, a grey tone image may be represented digitally by a digital image 
comprising pixels each of which has one pixel value representing the grey tone of the 
corresponding pixel. Similarly, a colour image may be represented by a digital image 
15 comprising pixels each of which have three pixel values, one for each of the colours red, 
green, and blue. 

Typically, a digital image is created by transmission of light towards an object and 
detection by an electronic camera of light reflected from or transmitted through the object. 

20 However, in general a digital image may be created by transmission of any kind of 
radiated energy, such as electromagnetic radiation, such as visible light, infrared 
radiation, ultraviolet radiation, X-rays, radio waves, etc, ultrasound energy, particle 
energy, such as electrons, neutrons, etc, etc, towards an object for interaction with the 
object and by detection of energy having interacted with the object, such as by reflection, 

25 refraction, absorption, etc. 

A digital image may be formed by any imaging system, such as radiometer systems, 
infrared systems, radar systems, ultrasound systems, X-ray systems, electronic cameras, 
digital scanners, etc, adapted to detect the kind of energy in question and to generate a 
30 digital image based on the energy detection. 

The amount of energy needed to record a digital image with a desired signal to noise 
ratio, i.e. the sensitivity of the imaging system, is determined by the noise level in the 
imaging system. Energy detectors of any kind generate noise that adds to the signal 
35 desired to be recorded. The signal to noise level of a digital image is typically required to 
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be comparable to if not better than the signal to noise level of an image recorded on a 
photographic film for subsequent reproduction in professional publications. 

Typically, an image recording system operating in the visible light range of 
5 electromagnetic radiation, such as an electronic camera, a digital camera, an electronic 
scanner, a digital scanner, etc, uses a solid state imaging device, typically a charge — 
coupled device (CCD), for recording of an image. 

The CCD is an array of a large number of light sensitive detectors connected to each 
1 0 other as an analog shift register. In each detector of the CCD a charge is formed that is 
proportional to the light energy incident on the detector during an integration period. The 
analog charge of each detector is shifted serially out of the CCD and is typically converted 
to a digital value whereby a digital representation of the recorded image is formed. Each 
pixel value of the digital image is equal to the digitised charge of the corresponding CCD 
15 detector. The pixel values may be transferred to an external computer through a computer 
interface or may be stored on a memory card or on a rotating magnetic recording medium. 

It is known in the art to use linear CCDs in an image recording system in which the linear 
CCDs light-sensing detectors are arranged in a single line. Typically, the array is moved 
20 across the image, scanning it one line at a time. For colour images, filters can be placed 
in front of the array, which then makes three passes across the image during image 
recording. 

Offering a good compromise of image resolution (high pixel count) and cost in an image 
25 recording system is the trilinear CCD array, which comprises three linear CCDs positioned 
side by side. Each line is covered by its own colour filter. The array makes a single pass 
across the image, and each linear CCD records a line at a time. 

At a higher cost, a two-dimensional CCD array can capture the entire image in a very 
30 short time. Three separate exposures with three colour filters can be used to make a 
colour representation. For example, the colour filters can be placed in a rotating wheel 
that sequentially inserts each colour filter in the optical path between the image forming 
optics of the image recording system and the two-dimensional CCD array. 
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3 

In the present context, the term exposure is to be understood in a broad sense as the time 
period during which an energy sensor is actually sensing the energy. For example, a 
photographic film is exposed to light whenever light is incident upon it, while a CCD is 
exposed to light when the elements of the CCD are allowed to integrate light incident 
5 upon them. The CCD is not exposed when its light sensing elements are short-circuited 
although light may be incident upon them. — 

It is well known in the art to size a digital image, i.e. to change, typically reduce or enlarge, 
the size of the digital image, i.e. the number of pixels of the digital image in order to 

10 minimise the amount of pixel data to be stored for later processing of the digital image. 
Various, sometimes complex, strategies for forming sets of pixels from which new pixel 
values are calculated may be employed. In a very simple example, a digital image may be 
downsized by reducing the number of pixels of the digital image by an integer, e.g. by a 
number of four. In this case, the pixels of the original digital image is divided into sets of 

15 four pixels each and each set of pixels is transformed into a new pixel of a pixel value 
equal to the average value of the original pixel values. Thereby, a new digital image of 
one-fourth the size of the original digital image is generated. 

Further, it is well-known to adjust brightness of an image produced from a digital image by 
20 forming a new digital image with the same number of pixels as the original image in which 
each of the new pixel values is generated by a linear or non-linear transformation of the 
corresponding original pixel value. 

Further it is well known from e.g. US 5,373,322 that in order to generate a high quality 
25 colour information relating to at least three primary colours are required at the position of 
each pixel in an image. In case where information's relating to three primary colours are 
not available in each pixel of e.g. a CCD interpolation complex methods involving linear or 
non-linear transformation methods are required. 

30 A disadvantage of these complex conventional processing methods is related to the time 
required for processing the image. To obtain a high quality colour image complicated 
mathematical algorithms are applied in the process of transforming a low-resolution colour 
image into a high-resolution colour image. 
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A further disadvantage of the conventional processing methods is related to the presence 
of colour artefact (aliasing) which significantly disturbs the interpretation of a colour image. 

It is an object of the present invention to provide a method and a system for estimating the 
5 missing colours in images recorded using colour filter array based CCD technology. 

It is another object of the present invention to provide a method and a system for esti- 
mating the missing colours in images recorded using scanner-based technology. 

10 It is a still another object of the present invention to provide an image processing method 
comprising the steps of 

(a) dividing an image into a matrix of pixels for holding colour values, and 

15 (b) interpolating a colour (A) value of a selected pixel by calculating the colour (A) 

value from colour values of a selected set of pixels that are positioned adjacent to 
the selected pixel, the influence of at least one pixel abutting the selected pixel 
being significantly reduced whereby blurring of distinct features of the image is 
substantially avoided. 

20 

By adjacent is meant that the selected set of pixels should be near the selected pixel. The 
term significantly reduced should be understood as lowly weighted or excluded from the 
selected set of pixels. 

25 The image processing method further comprising the steps of 

(a) forming a first set of pixels that are positioned adjacent to the selected pixel, and 
calculating a first variation value of colour values of the pixels of the first set, 

30 (b) forming a second set of pixels that are positioned adjacent to the selected pixel, 

and calculating a second variation value of colour values of the pixels of the second 
set, 

(c) comparing the first and second variation values, and calculating the colour (A) 
35 value from colour values of the set of pixels with the lowest variation value. 
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The image processing method further includes a step, wherein colour (B) values of the set 
of pixels with the lowest variation value are included in the calculation of the colour (A) 
value. In particular the colour (B) values are included in the calculation of variation values 
of the first and second set of pixels, respectively. 

5 

According to the present invention each of the steps of calculating variation values of the 
first and second set of pixels, respectively, comprises 

(a) calculating the ratio between colour (A) values and colour (B) values of abutting 
1 0 pixels of the respective set of pixels, 

(b) calculating a mean value of the calculated ratio values, and 

(c) calculating the variation value of the respective set of pixels as the sum of the 
1 5 absolute value of the difference between each of the ratios and the mean value. 

The step of calculating the colour (A) value comprises 

(a) calculating a weighted sum of colour (B) values of the set of pixels with the 
20 lowest variation value, 

(b) calculating a weighted sum of colour (A) values of the set of pixels with the 
lowest variation value, and 

25 (c) calculating the colour (A) value by multiplying the colour (B) value of the selected 

pixel with the ratio between the weighted sums of colour (A) and colour (B) values. 

The colour (A) value may be calculated from not only two colour values. The colour (A) 
value may be calculated from an arbitrary number of colour values. 

30 

The image recorded for further processing may be recorded by an electronic camera 
comprising a scanner or a CCD or a Bulk Charge Modulated Device (BCMD). 
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In case of an electronic camera comprising a CCD the camera may further comprise a 
colour filter mask positioned in front of the CCD, said colour filter mask comprising a plu- 
rality of colour filters. The colour filter mask may be of the type Bayer 2G. 

5 The selected set of pixels may take any form and may comprise an arbitrary number of 
pixels. — 

In particular, the selected sets of pixels may be arranged in rows and columns parallel to 
the rows and columns of the pixels in the image matrix. Alternatively the selected sets of 
10 pixels may be arranged in rows at an angle of approximately 45 degree relative to the 
rows and columns of pixels in the image. 

In another alternative the selected sets of pixels may be arranged in rows and columns 
parallel to the rows and columns of the pixels in the image matrix and at an angle of ap- 
15 proximately 45 degree relative to the rows and columns of pixels in the image. 

The selected set of pixels may also be arranged at an arbitrary angle relative to the rows 
and columns of the pixels in the image. 

20 The number of selected sets of pixels may be arbitrary. Preferably the number of selected 
sets of pixels is larger than 2, such as 4. 

It is a still another object of the invention to provide an image processing system 
comprising 

25 

(a) a memory for storage of colour values of a matrix of pixels of an image, and 

(b) a processor that is connected to the memory and that is adapted for interpolating 
a colour (A) value of a selected pixel by calculating the colour (A) value from colour 

30 values of a selected set of pixels that are positioned adjacent to the selected pixel, 

the influence of at least one pixel abutting the selected pixel being significantly 
reduced whereby blurring of distinct features of the image is substantially avoided. 

According to the invention the image processing system may further comprise an elec- 
35 tronic camera comprising a scanner and CCD. 
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It is an advantage of the present invention is that the method and system significantly im- 
proves the ability to estimate the missing luminance information in a digitised image. A 
correct estimation of the missing luminance information is vital for the impression of 
sharpness and detail richness. 

5 

It is further advantage of the present invention is that all chrominance information are pfe- 
served to give images the highest degree of colour fidelity while avoiding all of the well 
known aliasing. 

10 It is a still further advantage of the present invention is the relative low complexity of the 
method that makes it easy to implement in both software and/or hardware based applica- 
tions. The method is also applicable in other areas of imaging such as image scaling, im- 
age compression and visual enhancements such as unsharp masking and noise reduc- 
tion. 

15 

In the following the invention will be described with reference to the accompanying fig- 
ures. 

Fig. 1 shows a colour filter pattern of the type Bayer 2G. Each cell in the grid represents a 
20 photosite or pixel on the m x n array CCD where m is the number of photosites in each 
row on the CCD, n is the number of rows on the CCD, R is the number of photosites cov- 
ered with red filter material (called Red photosites), G is the number of photosites covered 
with green filter material (called Green photosites) and B is the number of photosites cov- 
ered with blue filter material (called Blue photosites) 

25 

Fig. 2 shows an illustration of four selected set of pixels. 

Fig. 3 shows a reference image recorded using a state-of-art digital camera with three 
CCD arrays. 

30 

Fig. 4 shows an image recorded using a digital camera with only one CCD array and 
processed according to the present invention. 

Fig. 5 shows an image recorded using a digital camera with only one CCD array and 
35 processed according to convention methods. 
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To construct a m x n RGB image the Green and Blue values have to be estimated at 
photosites covered with Red filter material and the Blue and Red values at photosites 
covered with Green filter material and finally the Red and Green values at photosites cov- 
ered with Blue filter material. 

5 

Note that the Bayer pattern has twice the amount of photosites covered with Green filter 
material than those with Blue and Red filter material, hence the name Bayer 2G. 

The part of the visual wavelengths sampled by the Green photosites represents most of 
10 the pure luminance information. The Blue and Red photosites carries most of the cromi- 
nance information, but the fact that the Blue and Red photosites also carries a lot of pure 
luminance information is crucial for the interpolation method of the present invention. 
In the present invention the missing Green information at photosites covered with Red 
filter material is solely based on the variation between values from a selected set of Green 
15 and Red photosites taken from the surrounding area. The set of values from photosites 
must be segmented in such a way that it is possible to distinguish between edge bounda- 
ries. 

Selection of sets of pixels to be considered for possible interpolation depends mostly on 
20 the actual layout of the colour filter array in question. The number and the choice of set of 
pixels have profound impact on the performance of the method. 

In the case of the Bayer 2G pattern a set of pixels for estimating the Green samples at 
Red photosites could be as shown in Fig. 2. 

25 

The same principle can be applied when selecting the set of pixels for estimating the 
Green samples at Blue photosites. Again, the set of photosites that are chosen for each 
set of pixels is critical for the correctness of the estimation. 

30 Referring to Fig. 2, the 4 sets of pixels appears to be sufficient for detection of very fine 
details still without introducing too much noise and other artefacts in low detail areas of 
the image. 

A variation value, corr, is calculated for each set of pixels. The variation function ex- 
35 presses to which extend the information reduced from one colour at a given site can be 
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used to estimate the influence of the same colour at a different site, i.e. to reduce the in- 
fluence of the Green colour at e.g. the photosite of a Red filter. 

Due to performance constrains the variation formula must be kept as simple as possible 
5 still without sacrificing the quality of the output. The formula can be expressed using clas- 
sic linear correlation. — 

The formula used in a preferred embodiment of the present invention is given by: 



10 

Where G is the green pixels in the set of pixels, R is the red pixels in the set of pixels, m is 
the number of pixels in each set of pixels, mean is the average ratio between red and 
green pixels and corr is the variations between red and green pixels. Lower values repre- 
sent higher correlation. 

15 

When a variation value has been calculated for every set of pixels, the set of pixels with 
the highest correlation (lowest numeric value) is selected. This variation value will be used 
in the further estimation process. 

20 The missing colour at a given photosite is expressed in terms of the information contained 
in the selected set of pixels. If the variation is below a certain threshold, due to noise or 
other factors, an alternative set of pixels using a different variation method may be se- 
lected. 



mean 





25 



The missing colour is estimated as a weighted sum of the green samples divided by the 
weighted sum of the red samples multiplied by the sampled value from the photosite to be 
estimated. This may be expressed in the following way: 
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Rw 



- S v. 

n= 1 



n= 1 



* Rw 

5 where Rw is the weighted sum of Red samples in the set of pixels, Gw is the weighted 
sum of Green samples in the set of pixels, Rk is the sampled value from the photosite in 
question and Ge is the estimated Green value. 

In order to process the full image the above-mentioned process is repeated for each pixel 
10 in the image until all the missing colours have been estimated. 
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CLAIMS 

1. An image processing method comprising the steps of 

5 - dividing an image into a matrix of pixels for holding colour values, and 

- interpolating a colour (A) value of a selected pixel by calculating the colour (A) 
value from colour values of a selected set of pixels that are positioned adjacent to 
the selected pixel, the influence of at least one pixel abutting the selected pixel 

10 being significantly reduced whereby blurring of distinct features of the image is 

substantially avoided. 

2. A method according to claim 1 , further comprising the steps of 

15 - forming a first set of pixels that are positioned adjacent to the selected pixel, 

- calculating a first variation value of colour values of the pixels of the first set, 

- forming a second set of pixels that are positioned adjacent to the selected pixel, 

20 

- calculating a second variation value of colour values of the pixels of the second 
set, 

- comparing the first and second variation values, and 

25 

- calculating the colour (A) value from colour values of the set of pixels with the 
lowest variation value. 

3. A method according to claim 2, wherein colour (B) values of the set of pixels with the 
30 lowest variation value are included in the calculation of the colour (A) value. 

4. A method according to claim 2 or 3, wherein colour (B) values are included in the 
calculation of variation values of the first and second set of pixels, respectively. 
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5. A method according to claim 4, wherein each of the steps of calculating variation values 
of the first and second set of pixels, respectively, comprises 

- calculating the ratio between colour (A) values and colour (B) values of abutting 
5 pixels of the respective set of pixels, 

- calculating a mean value of the calculated ratio values, and 

- calculating the variation value of the respective set of pixels as the sum of the 
10 absolute value of the difference between each of the ratios and the mean value. 

6. A method according to any of the preceding claims, wherein the step of calculating the 
colour (A) value comprises 

15 - calculating a weighted sum of colour (B) values of the set of pixels with the lowest 

variation value, 

- calculating a weighted sum of colour (A) values of the set of pixels with the lowest 
variation value, and 

20 

- calculating the colour (A) by multiplying the colour (B) value of the selected pixel 
with the ratio between the weighted sums of colour (A) and colour (B) values. 

7. A method according to claim 1 , wherein the colour (A) value is calculated from an arbi- 
25 trary number of colours. 

8. A method according to claim 1 , wherein the number of selected sets of pixels is arbi- 
trary. 

30 9. A method according to claim 1 , wherein the image is recorded by an electronic camera. 

10. A method according to claim 9, wherein the electronic camera comprises a scanner. 

1 1. A method according to claim 9, wherein the electronic camera comprises a C-MOS 
35 imager. 
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12. A system according to claim 9, wherein the electronic camera comprises a BCMD. 



13. A method according to claim 9, wherein the electronic camera comprises a CCD. 

5 14. A method according to claim 13, wherein the electronic camera further comprises a 
colour filter mask positioned in front of the CCD, said colour filter mask comprising a plu- 
rality of colour filters. 

15. A method according to claim 14, wherein the colour filter mask is of the type Bayer 
10 2G. 

16. A method according to claim 1 , wherein the selected sets of pixels are arranged in 
rows and columns parallel to the rows and columns of the pixels in the image. 

15 17. A method according to claim 1, wherein the selected sets of pixels are arranged in 
rows at an angle of approximately 45 degree relative to the rows and columns of the pix- 
els in the image. 



18. A method according to claim 1 , wherein the selected sets of pixels are arranged in 
20 rows at an angle of approximately 45 degree relative to the rows and columns of the pi- 
xels in the image and in rows and columns parallel to the rows and columns of the pixels 
in the image. 

19. A method according to claim 1, wherein the selected sets of pixels are arranged in 
25 rows at an arbitrary angle relative to the rows and columns of the pixels in the image. 

20. A method according to claim 1, wherein the selected set of pixels may take any form. 

21 . A method according to any of the claims 16-20, wherein the number of selected set 
30 of pixels is 4. 



PCT/DK99/0Q393 

14 

22. An image processing method comprising the steps of 

- dividing an image into a matrix of pixels for holding colour values, and 

5 - interpolating a colour (A) value of a selected pixel by calculating the colour (A) 

value from colour values of a selected set of pixels that are positioned adjacent ttr 
the selected pixel, the influence of at least one pixel abutting the selected pixel 
being significantly reduced whereby blurring of distinct features of the image is 
substantially avoided. 

10 

23. A method according to claim 22, further comprising the steps of 

- forming at least two set of pixels each of these sets being positioned adjacent to 
the selected pixel, 

15 

- calculating a variation value of colour values of the pixels of each set, 

- comparing each of the calculated variation values, and 

20 - calculating the colour (A) value from colour values of the set of pixels with the 

lowest variation value. 

24. A method according to claim 23, wherein colour (B) values of the set of pixels with the 
lowest variation value are included in the calculation of the colour (A) value. 

25 

25. A method according to claim 23 or 24, wherein colour (B) values are included in the 
calculation of variation values of each set of pixels.26. A method according to claim 25, 
wherein each of the steps of calculating variation values of each set of pixels, comprises 

30 - calculating the ratio between colour (A) values and colour (B) values of abutting 

pixels of the respective set of pixels, 
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- calculating a mean value of the calculated ratio values, and 
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- calculating the variation value of the respective set of pixels as the sum of the 
absolute value of the difference between each of the ratios and the mean value. 

27. A method according to any of the claims 22 - 26, wherein the step of calculating the 
5 colour (A) value comprises 

- calculating a weighted sum of colour (B) values of the set of pixels with the lowest 
variation value, 

10 - calculating a weighted sum of colour (A) values of the set of pixels with the lowest 

variation value, and 

- calculating the colour (A) by multiplying the colour (B) value of the selected pixel 
with the ratio between the weighted sums of colour (A) and colour (B) values. 

15 

28. A method according to claim 22, wherein the colour (A) value is calculated from an 
arbitrary number of colours. 

29. A method according to claim 22, wherein the number of selected sets of pixels is arbi- 
20 trary. 

30. A method according to claim 22, wherein the image is recorded by an electronic cam- 
era. 

25 31 . A method according to claim 30, wherein the electronic camera comprises a scanner. 

32. A method according to claim 30, wherein the electronic camera comprises a C-MOS 
imager. 

30 33. A method according to claim 30, wherein the electronic camera comprises a CCD. 
34. A method according to claim 30, wherein the electronic camera comprises a BCMD. 
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35. A method according to claim 33, wherein the electronic camera further comprises a 
colour filter mask positioned in front of the CCD, said colour filter mask comprising a plu- 
rality of colour filters. 

5 36. A method according to claim 35, wherein the colour filter mask is of the type Bayer 
2G. - 

37. A method according to claim 22, wherein the selected sets of pixels are arranged in 
rows and columns parallel to the rows and columns of the pixels in the image. 

10 

38. A method according to claim 22, wherein the selected sets of pixels are arranged in 
rows at an angle of approximately 45 degree relative to the rows and columns of the pix- 
els in the image. 

15 39. A method according to claim 22, wherein the selected sets of pixels are arranged in 
rows at an angle of approximately 45 degree relative to the rows and columns of the pix- 
els in the image and in rows and columns parallel to the rows and columns of the pixels in 
the image. 

20 40. A method according to claim 22, wherein the selected sets of pixels are arranged in 
rows at an arbitrary angle relative to the rows and columns of the pixels in the image. 

41 . A method according to claim 22, wherein the selected set of pixels may take any form. 

25 42. A method according to any of the claims 37 - 41 , wherein the number of selected sets 
of pixels is larger than 2, such as 4. 

43. An image processing system comprising 

30 - a memory for storage of colour values of a matrix of pixels of an image, and 

- a processor that is connected to the memory and that is adapted for interpolating a 
colour (A) value of a selected pixel by calculating the colour (A) value from colour 
values of a selected set of pixels that are positioned adjacent to the selected pixel, 
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the influence of at least one pixel abutting the selected pixel being significantly 
reduced whereby blurring of distinct features of the image is substantially avoided. 

44. A system according to claim 43, comprising an electronic camera. 

45. A system according to claim 44, wherein the electronic camera comprises a scanner. 

46. A system according to claim 44, wherein the electronic camera comprises a C-MOS 
imager. 

47. A system according to claim 44, wherein the electronic camera comprises a CCD. 

48. A system according to claim 44, wherein the electronic camera comprises a BCMD. 
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